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Abstract 

Through use of multiband {U, Bj, Rp, In) photometry we have isolated high redshift 
(3.0<z<3.5) galaxy candidates in a survey of 1.27 deg^ to Rp — 21.25 and a survey of 0.02 
deg^ to Rp — 23.5. Our pool of candidates constrains the nature of the 3.0 < z < 3.5 
luminosity function over the range L* <^ L <^ lOOL*. if we grant a similar level of 
completeness to these data as for very faint samples (to R = 25.5) selected in a similar 
fashion. Our constraints agree with the high redshift sky density at Rp = 20.5 estimated 
from Yee et al. 's (1996) serendipitous discovery of a bright, z = 2.7 galaxy, as well as the 
density at Rp ~ 23 by Steidel et al. (1996b). We strongly rule out - by more than two 
orders of magnitude at Mrj, — —25 - the L > L* luminosity function for 2; = 3 — 5 galaxies 
obtained by a photometric redshift analysis of the Hubble Deep Field (HDF) by Gwyn & 
Hartwick (1996). Our results at Rp ~ 23 are more consistent with the photometric redshift 
analysis of the faint HDF galaxies by Sawicki & Yee (1996), but our present upper limits at 
the brightest magnitudes {Rp < 21.5, Mji^ < —24) allow more generous volume densities 
of these super-L* galaxies. 

Subject headings: galaxies: luminosity function - galaxies: evolution - galaxies: 
distances and redshifts 
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1. Introduction 

When deep imaging surveys revealed a significant population of blue objects (well in 
excess of no-evolution models) at S > 23, it was initially thought (Tyson 1988, Cowie 
1988, Cowic et al. 1988) that this could be the signature of "primeval galaxies" (PGs) - 
counterparts to present day ~ L* galaxies undergoing an initial, extremely bright burst of 
star formation at high z (Partridge & Peebles 1968). Ever deeper redshift surveys (now 
B ~ 24, Songaila et al. 1994, Glazebrook et al. 1995), however, revealed galaxies only to 
z < 0.8, and it became evident that galaxies with redshifts as high as 3 were not likely to 
represent a substantial fraction of the galaxies, even to S ~ 25 (Koo et al. 1996). 

The conspicuous paucity of faint, high z galaxies had already been shown by two 
studies of faint galaxy colors. At 2.7 < z < 3.4, galaxies should exhibit a particularly red 
U — B, compared to a rather blue color at longer wavelengths, because of the presence of the 
Lyman hmit in the U passband. The first apphcation of this test by Koo, Kron & Majewski 
(see Majewski 1988, 1989) demonstrated the number oi B < 24.5 galaxies showing the 
expected z > 3 color signature to be negligible - < 1%. With deeper data, Guhathakurta, 
Tyson & Majewski (1990) showed that the number of galaxies to S ~ 27 showing the 
expected z > 3 color signature was no more than 7%, and likely < 1% of galaxies. 

More recently, Steidel et al. (1995 and references therein, "S95") have repeated the 
"Lyman limit imaging" experiment over ~ 0.03 deg^ to their 3? = 25.5. They confirm 
the relatively low surface-density of high z candidates, and with the Keck 10-m have 
spectroscopically verified 22 of 37 color-selected candidates are indeed at 3 < 2; < 3.5 
(Steidel et al. 1996b, "S96"). All galaxies in S96 have 9? =23.7-25.5, implying luminosities 
near present day L* and slightly brighter. From their data, S96 estimate the comoving 
space density of these objects to be approximately 1/2 that of present day L > L* galaxies. 
Both Steidel et al. (1996a) and Lowenthal et al. (1997) find comparable results in the same 
magnitude and redshift range in the Hubble Deep Field (HDF), despite more liberal color 
selection in the latter survey. 

The numbers of objects much brighter than L* is less well constrained. Yee et al. 
(1996) have discovered serendipitously a "normal" (i.e., neither AGN nor radio), V — 20.5 
galaxy eit z — 2.7. Though super-luminous at Mr — 5 log h —25 (qo=0.5, h — Hq/IOO), 
this galaxy is spectroscopically similar to the S96 galaxies. Based on one galaxy in their 
0.66 deg^ survey, Yee et al. estimate the density of such objects at i? ~ 20 is 10°=*=^ deg~^. 

Meanwhile, Gwyn & Hartwick (1996, "GH") attempted to determine photometric 
redshifts for galaxies in the HDF and claim dramatic changes in the galactic luminosity 
function ($(M)) from < 2; < 5 with $(M) becoming flat between -24 < Mb < -15 for 
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3 < z < 5. They predict a substantial abundance of galaxies up to M* — 4 at high z. In stark 
contrast, the photometric redshift analysis of the HDF by Sawicki et al. (1996, "SLY") finds 
a more prosaic z = 3 — 4 luminosity function, adequately described by a Schechter function 
with a = —1.3 and (p* = 0.023/i^ Mpc~^. An extrapolation of this function predicts many 
orders of magnitude less high z galaxies at Rp = 21 than GH. Yet another photometric 
analysis of the HDF by Mobasher et al. (1996) suggests strong luminosity function evolution 
to z — 3, and imphes numbers of bright z — 3 galaxies intermediate between GH and SLY. 
While all of these groups suggest that in the HDF they are seeing the formation of L > L* 
galaxies at high z, there seem to be vast differences in the implied nature of the luminosity 
function, especially for bright galaxies. 

The range in HDF results may be attributable to the substantial uncertainty in the 
application of photometric redshifts at very faint magnitudes. Unlike the photometric 
redshift study of brighter galaxies by GonnoUy et al. (1996), at HDF depths there is no 
adequate spectroscopic training set available for cahbration. While S96 have a handful 

of spectra of z > 3 galaxies, the vast majority of objects to 9? ~ 25 and beyond are 
without spectroscopic redshifts. Most troubling to the interpretation and application of 
spcctro-photometric galaxy models is the near-degeneracy in color between particular 
redshifts (e.g., at < 2; < 1 for the bluest galaxies and z ~ 2.5 for all galaxies; see figure 1); 
this plausibly produced the apparent strong bimodality in the redshift distribution inferred 
by GH. Unlike in GonnoUy et al. , GH and SLY do not use apparent brightness to break 
such degeneracies in the redshift estimates. Moreover, as pointed out by SLY, differences in 
the model spectral energy distributions (SEDs) - particularly in the still poorly understood 
rest-frame ultraviolet where internal reddening and intervening absorption are important - 
lead to substantially different results. It is important, therefore, to check the HDF results, 
especially at magnitudes where spectroscopic confirmation is feasible. 

Apart from the S96 data near L* and the loose Yee et al. (1996) constraint, there is 
scant spectroscopic redshift data to explore the nature of the luminosity function of 2; > 3 
galaxies. Ironically, it is <I>(M) at magnitudes brighter than those explored by S96 that is 
most poorly defined due to a lack of reliable data; if the high redshift <I>(A'f) is of prosaic 
form, with a steep decline toward the bright end, much larger survey areas arc required 
to explore the L » L* domain than have been achieved with CCD surveys to date. 
We have undertaken a large area, photometric search for bright, high z galaxies. Even 
without spectroscopy, we constrain the high z luminosity function based on the magnitude 
distribution of our high z candidates. To do so, we rely on the good correspondence between 
high z galaxy candidates identified by S95 through similar selection criteria and bona fide 
high z galaxies among these candidates as confirmed by S96. 
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2. High Redshift Galaxy Search 



Our search for bright, z > 3.0 galaxies utilizes two sets of multicolor galaxy catalogues. 
The first data set consists of the photographic catalogues generated for the Kitt Peak 
Galaxy Redshift Survey (KPGRS; Munn et al. 1997) and faint quasar surveys (Kron et 
al. 1992). These catalogues cover four separate regions of the sky totaling 1.27 deg^ with 
photometry from sky-limited, Mayall 4-m photographic plates in the UBjRfIn passbandsQ 
calibrated with deep CCD photometric sequences (e.g. Majewski et al. 1994). While these 
catalogues reach to Rp = 23, we choose here a conservative catalogue limit of Rp = 21.25, 
where random errors in Bj, Rp and In are at most 0.3 mag (smaller than the color 
difference between our z > 3 selection thresholds and the locus of low z galaxies). 

The panels in Figure 1 show the progression oiU — Bj, Bj — Rp and Rp — In galaxy 
colors with redshift. The iso-2; loci for different galaxy types were generated with Bruzual 
& Chariot's (1993) models for a range of star-formation histories^, plus observed elliptical 
and starburst (using the galaxy N4449) spectral energy distributions (SEDs). Model and 
observed SEDs were reddened as a function of band and redshift to account for intervening 
absorption (only), as prescribed by Madau (1995). This process is identical to that in S96, 
except that we include a broader range of SEDs. For colors of 2 > 3.0 galaxies this has little 
consequence, however it is important at lower redshifts. Based on our models, dashed lines 
delimit the region of each color-color diagram inhabited by 2: > 3.0 galaxies. To justify this 
selection, we show (top-left panel. Figure 1) the S96 z > 3.0 galaxies in our UBjRp system 
using the transformations in Majewski (1992) and Steidel & Hamilton (1993); the symbols 
are coded for those objects S96 classified as "robust" ([/-band drop-outs falling within the 
selection boundary based on 1-a limits), and "marginal" (non- "robust" objects within the 
selection boundary). We also show (top-right panel, Figure 1) all known 2.7 < z < 3.0 and 
;z > 3.0 QSOs in our fields (Kron et al. 1992). 

The middle and right panels of Figure 1 show our Rp < 21.25 sample. We are 
interested in setting upper limits on the numbers of bright, high z galaxies; our selection 
algorithm reflects a liberal acceptance threshold that sets a conservative upper limit on 
$(M) while maintaining reasonable reliability. As a first acceptance criterion, we adopt a 



''Our U band is virtually identical to the standard photoelectric U (Koo 1985). Steidel & 
Hamilton's (1993) f/„ is about 100 A bluer, and UpsQow for the HDF is about 700 A bluer. 

^ Colors are for evolving and non-evolving model galaxies with 16 Gyr ages at ^ = 
(qo=0.1, Ho=50 km s^^ Mpc~\ A=0), Salpeter initial mass, and 0.01 < /i < 0.95, where 
is the fraction of galactic mass in stars after 1 Gyr of star formation. 
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similar selection function to S96 - objects with red U — Bj and blue Bj — Rp colors, as 
illustrated in the top row of Figure 1. However, our models show the U — Bj, R^- — Ij^ 
diagram (middle row. Figure 1) affords a much cleaner separation of high z galaxies from 
the low z locus. We accept high z galaxy candidates from this diagram as well. Galaxies 
selected in the {U — Bj, Bj — Rp) and {U — Bj, Rp — In) diagrams need not be the same. 
For example, a high z galaxy might be missed in the U — Bj, Bj — Rp plane if the line 
of sight to that object passes through a sufficient number of neutral hydrogen clouds for 
significant suppression of the observed Bj flux (with both U — Bj and Bj — Rp affected). 

The Bj — Rp, Rp — /jv diagrams in the bottom panels of Figure 1 reveal the high z locus 
is not as well separated as in the other diagrams. As a compromise between completeness 
and low z contamination (reliability), we (1) adopt a more conservative color cut, but 
(2) accept galaxies in this diagram only if they are bona fide U band drop-outs. Objects 
selected in this way satisfy the relevant Rp — In color criterion in the {U — Bj, Rp — In) 
diagram, but have U — Bj upper limits insufficient to place them confidently within the 
z > ?) region. These objects are faint in Bj, so their exclusion in the middle panel is likely 
due only to the magnitude limit of the U plates. Hence these are plausible z > "i candidates. 
In the Bj — Rp, Rp — In diagrams we have the potential to discover galaxies at redshifts 
even higher than 3.5, yet no such "Sj-band drop-outs" were found. 

Each high z candidate was inspected visually on a number of photographic plates to 
ensure reliability. We find twelve resolved (unlikely to be either stars or QSOs) z > 3.0 
candidates between 19.25 < Rp < 21.25. We also find 20 unresolved sources with 
Rp < 21.25; three are spectroscopically identified as z > 2.9 QSOs and two as stars in our 
QSO survey (Kron et al. 1992). No galaxies at 2; > 1 have been identified among any of the 
QSO candidates in Kron et al. (1992). 

To bridge our study of the high z luminosity function from Rp = 21.25 to the very 
deep S96 sample, we have generated deeper images in two 39 arcmin^ subficlds of SA 57 by 
stacking PDS microdensitometer plate scans of five U plate images, ten Bj plate images and 
five Rp plate images (see Majewski 1988). Candidate z > 3 galaxies are selected here only 
on the basis oi U — Bj, Bj — Rp as before (no comparably deep In image was available). A 
total of 11 candidates (both stellar and nonstcllar) are found (small triangles in top-middle 
panel of Figure 1) to the conservative hmit of Rp — 23.5. 



3. High Redshift Luminosity Function 
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Our z > 3 galaxy candidates represent the highest possible density of bright, S96-like 
galaxies if we assume a similar level of completeness in our sample as has been assumed 
for S96. We believe no 2; > 3 galaxies lie within our unresolved sample to Rp < 21.25, 
but, in the spirit of upper limits, we include discussion of this sample here. Our survey 
should be more complete than, for example, S96 since we utilize multiple combinations of 
colors. If there exists a population of high z galaxies not chosen by our selection criteria, 
our comparison to other surveys using similar S96-like selection criteria is still valid. 

In Figure 2 we compare the various studies of the ^ > 3 luminosity function in a 
cosmologically model-independent way: 

(i) The luminosity functions of GH (in the range 3 < 2; < 5) and SLY (in the range 
3 < z < A) are transformed into the apparent differential counts A{Rf) for galaxies lying 
in the redshift shell between 3 and 3.5, the range of redshift to which our data apply. To 
do this we have adopted their respective cosmologies to scale by the appropriate volumes 
and luminosity distances, and assumed A;-corrections for N4449. The observed spectrum of 
N4449 has been extended below 1250 A using the best-matching Bruzual & Chariot model 
in the range 1250-2000 A. Note that GH and SLY calculate "the z>3 luminosity function" 
beyond our z — 3.5 limit, yet both studies find the space density falling rapidly beyond 

z 2. Thus our estimation of the predicted A(Rf) for their $(Mr^) in the lowest redshift 
shell of their broader z ranges provides lower limits on the counts. This is particularly 
relevant to the gross discrepancy between our derived upper limits to A[Rf) and the GH 
results (presented as lower limits) detailed below. 

(ii) Candidate z > 3 galaxies at faint magnitudes were compiled from S95 and S96 in 
two ways, a) In S95, four of five fields had well-defined samples and areas (i.e. excluding 
QOOOO-263), yielding 15 candidates in 20.7 arcmin^ to 3? < 25.5 defined as "robust" by them. 
From S96, we counted candidate and confirmed z > 3 galaxies in two fields (QOOOO-263 and 
SSA 22) to the same depth in a total area we estimate to be 45.4 arcmin^. These tallies 
exclude spectroscopically confirmed stars, QSOs, or galaxies at z < 3. A total of 36 robust 
candidates in 66.1 arcmin^ are counted, 22 of which are spectroscopically confirmed. Based 
on the 81% reliability of spectroscopically identified candidates (3 of 16 are QSOs) we scaled 
the remaining unconfirmed robust candidates to derive number counts in the Rp band for 
z > 3 galaxies, b) For S95, we have counted, in the same 4 of 5 fields, additional objects 
not considered "robust" but still within the color region believed to contain 3 < ^ < 3.5 
galaxies. This yielded 23 candidates (including "robust" ones) in 20.7 arcmin^. 

(iii) We include the z ~ 2.7 Yee et al. as a datum at Rp ~ 20.35 assuming one source 
in their survey area at this apparent brightness. 
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Without redshifts, we constrain the density of resolved z > 3.0 galaxies to < 10 ± 3 
deg~^ at Rp < 21.25, or < 22 ± 4 deg~^ if we include unresolved candidates (minus 
the known QSOs and stars). Our counts are in strong conflict - at least two orders of 
magnitude - with GH at this depth. We note that the counts of z > 3 galaxies in the GH 
analysis rivals our total galaxy counts for Rp < 22 (Figure 2), although if we had adopted 
redder /c-corrections, the results of the surveys would be in somewhat better agreement. At 
22 < Rp < 23.5, our hmits are similar to the upper limits of the more liberal set of the 
fainter S96 candidates ([ii]b above). The combination of the S96 upper limits and our limits 
at brighter magnitudes suggests a rapid decline in the z > 3 luminosity function brighter 
than Mr - 5 log -20.5 to -21. This corresponds to the SLY et al. M* at 2; = 3.25 
[Rp — 24.5), or faO.75 mag brighter than the local galaxy luminosity function. In general, 
our upper limits and those derived from S95 agree with the SLY luminosity function for 
23.5 < Rp < 25. However, over the brighter range of our survey, 19 < Rp < 23.5, our 
upper limits allow for a much more gradual bright-end decline than suggested by the SLY 
extrapolation, and arc consistent with the Yee et al. (1996) serendipitous discovery. 

While the SLY luminosity function overestimates the numbers of robust candidates 
from S95 and S96, our more liberal selection from their data is marginally consistent to 
Rp 25. However, it is critical to discriminate between the SLY suggestion of a rising faint 
end of the luminosity function and the flatter faint end hinted at by the S96 data. Upper 
limits on the shape of the L < L* luminosity function could be checked via the f/-band 
drop-out method in a similar manner to what we have done here at brighter magnitudes. 
While the HDF data are appropriately deep for such an exercise, the Upsoow of WFPC2 is 
much bluer than ground-based, Johnson U; this increases the sensitivity of HDF data to 
z ~ 2.2 and complicates direct comparisons. Hence there is a need for ultra-deep imaging 
in the Johnson U band even over relatively small fields of view. 
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FIG. 1. - U — Bj, Bj — Rp, Rp — In color-color diagrams. Left panels: the locus 
of non-evolving observed, model, and passively evolving model galaxy colors (see key in 
middle- left panel) as a function of redshift (labels, connected by dotted lines of constant z)] 
our z > 3 color-selection (long-dashed lines); the ?> < z < 3.5 samples from S95 and S96, 
transformed to our photometric system (top-left); and the Yee et al. datum, transformed 
from their g,V,r,I bands using relations from Fukugita et al. (1995; bottom-left). Middle 
panels: our sample of all galaxies (points) and extended high z candidates to Rp — 21.25 
and high z candidates from stacks of photographic plate images to Rp — 23.5. Right panels: 
our sample of all stellar sources (points), stellar high z candidates to Rp — 21.25, and 
confirmed QSOs and stars. Keys in middle and right panels apply to all middle and right 
panels. Note the significant number of our candidates in the same region of color-space as 
S96's confirmed 3.0 < z < 3.5 sample. We select high z candidates, however, using all three 
color-color diagrams. 

FIG. 2. - Differential counts of candidate and confirmed 3 < z < 3.5 galaxies from 
figure 1, as described in text and key. Error bars enclose 68.3% confidence intervals (Gehrels 
1986). Also shown (lines) are counts for 3 < 2; < 3.5 galaxies from luminosity functions 
inferred from the HDF via photometric redshifts (GH and SLY). The dotted portion of 
SLY is an extrapolation of their best-fitting Schechter function. The absolute magnitude 
scale at z = 3.25 is shown at the top, while the change in Rp with redshift for constant 
luminosity and qo=0.5 is shown at the bottom. Total Rp galaxy counts from Kron (1980) 
and Majewski (1988) are shown for reference. 
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Figure 2 Bershady et al. 



